Maintaining the security of computer systems is an important and difficult problem. For a single computer system, system logs, firewalls, and other intrusion detection systems provide a certain level of security, but as computer systems and the attacks on them become more complex, detecting attacks on the system can become more difficult. For example, a complex computer system may include storage services, computing services, and virtual networking services that are shared across multiple customers as well as services and servers dedicated to individual customers. Attacks can be directed at any number of these systems, and a successful attack may be leveraged to compromise other connected services and subsystems. Therefore, detecting such attacks early is an important step in mitigating and preventing severe system compromise.
In addition, it is difficult to measure the effectiveness of various systems, such as intrusion detection systems, at preventing or mitigating attacks. Furthermore, attackers are continually developing new attack patterns to circumvent security measures. For example, attackers are continuously searching for exploits to launch various attacks, such as a zero day attack, that are not detected by an intrusion detection system. These zero day attacks/vulnerabilities refer to gaps in the security of a particular application that is unknown to the developer and therefore may not be secured or accounted for by the developer. Therefore, measuring the effectiveness of intrusion detection systems and other security measures is important to the development and improvement of those security measures.